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About the Study 


This issue brief is part of a five-part series that reports 
findings from research conducted by ACT in collaboration 
with Region One Education Service Center, a provider of 
educational services to over 430,000 students in South 
Texas who are primarily Latinx and from low-income 
communities. The research was funded by a NewSchools 
Venture Fund grant awarded to ACT’s Center for Social, 
Emotional, and Academic Learning and ACT’s Center for 
Equity in Learning. The overarching goal of the research was 
to gain a better understanding of factors related to social and 
emotional learning (SEL), including program efficacy, 
implementation, and family engagement. This brief focuses 
on the efficacy of SEL lessons implemented and includes 
results from surveys completed by students who participated 
in an SEL lessons during the 2019-2020 school year. The 
goal of the study was to better understand students’ 
reactions to SEL lessons, including whether they found the 
lessons engaging, whether they acquired knowledge and 
skills, and whether they were able to apply content from the 
lessons in and out of school. 


So What? 


Findings from the current study provide efficacy evidence 
for SEL lessons on Sustaining Effort, Getting 

Along With Others, and Maintaining Composure. Following 
the SEL lessons, learners reacted 

positively, perceived learning gains, and felt confident 
applying content from these lessons. Although additional 
research is needed, these initial results are promising 
especially given the disruptions the COVID-19 pandemic 
caused as the lessons were being implemented. 


Now What? 


These results suggest SEL programming may be an 
effective way to support learners who have been historically 
underserved and who often need additional supports 
because they face significant barriers to college entry and 
completion. As such, a promising direction for future 
research is examining SEL implementation with a broader 
sample of historically underrepresented groups of 

students. This research can help inform efforts to focus on 
the whole child in education while supporting diverse 
learners. 


ACT 


ACT.org/research 
R2104 
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Student Reactions and Perceived Gains to Social and 


Emotional Learning Lessons 


Social and emotional (SE) skills can be defined as interpersonal, self-regulatory, and task- 
related behaviors that are important for adaption to and successful performance in educational 
and workplace settings (Casillas, Way, & Burrus, 2015). These skills are related to important 
outcomes, including appropriate behavior, healthy relationships, and academic success (e.g., 
Domitrovich, Durlak, Staley, & Weissberg, 2017). Further, a growing literature suggests social 
emotional learning (SEL) interventions can effectively enhance SE skills and other important 
outcomes, including academic achievement (for meta-analyses, see Corcoran, Cheung, Kim, & 
Xie, 2018; Durlak, Weissberg, Dymnicki, Taylor, & Schellinger, 2011; Taylor, Oberle, Durlak, & 
Weissberg, 2017). 


SEL interventions may be especially beneficial for students from historically underrepresented 
groups such as students of color and students from low-income communities. These students 
face significant barriers to educational achievement, such as exclusionary discipline practices, 
higher levels of educator stress, lower academic expectations, and lower rates of college 
readiness (ACT, 2016; Gilliam, Maupin, Reyes, Accavitti, & Shic, 2016; Gregory, Skiba, & 
Noguera, 2010; McKown & Weinstein, 2008; Morris & Perry, 2016; Peng & Lee, 1994). 


To support a more equitable education, historically underrepresented groups can benefit from 
educational interventions designed with their needs in mind (Jagers, Rivas-Drake, & Williams, 
2019). For instance, Reyes, Elias, Parker, and Rosenblatt (2013) suggest SEL can help 
students learn to manage and communicate their emotional distress and conflict, which in turn 
can improve educator-student relationships. Additionally, Rivas-Drake, Lozada, Pinetta, and 
Jagers (2020) suggest that improvements in empathy and perspective-taking may underlie 
positive development, including identity development. However, more research is needed to 
directly examine the effects of SEL with students from historically underrepresented groups. To 
that end, the current study aimed to investigate student reactions to SEL lessons with a 
population of students from schools within Region One, a network of schools along the United 
States—Mexico border in South Texas serving 96% Latinx, 85% low-income, and 38% English 
learners. 


To support student growth in these important skills, students received supplemental SEL 
lessons following the Mosaic™ by ACT®: Social Emotional Learning assessment.’ These 
lessons are delivered through student reports. These lessons are designed to be used in 
universal settings to help students develop SE skills. Lessons are designed to support students’ 
understanding of and ability to demonstrate the skills of Sustaining Effort, Getting Along With 
Others, Maintaining Composure, Social Connection, and Keeping an Open Mind. For 
information on each of these skills, see Table 1. 


1 The lessons discussed in this report are available through the Mosaic by ACT: Social Emotional Learning 
rae leak via student Repos: od more information about the =e skills cali by these lessons, please visit 
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To test the efficacy of these lessons, we utilized ACT's Efficacy Framework (Mattern, 2019). This 
framework specifies various sources of efficacy data linked to Kirkpatrick’s (1976) evaluation 
model. This model organizes outcomes into four levels: 


* Level I: the extent to which learners find the lessons engaging and relevant 


* Level Il: the extent to which learners acquire intended knowledge and skills from the 
lessons 


* Level Ill: the extent to which learners apply content from the lessons 
* Level IV: the extent to which target outcomes are improved based on the lessons 


Murano, Walton, Burrus, Anguiano-Carrasco, and Way (2019) provided initial evidence of 
efficacy for a subset of these lessons. In this study, eighth-grade students from Arizona 
participated in a lesson designed to help students develop the skill of Sustaining Effort. This was 
followed by a survey asking students about their impressions of the lesson. Most students 
reacted positively, reporting that they enjoyed the lesson, found the content valuable, and 
thought they could use skills from the lesson throughout their whole lives. Additionally, most 
students felt confident that they could define Sustaining Effort as well as identify associated 
behaviors and describe its importance. These results provide evidence of efficacy for both Level 
| and Level II of Kirkpatrick’s model, but further research is needed, particularly on lessons 
targeting other skills. 


To gather further efficacy evidence, the current study examined student responses to lessons on 
Sustaining Effort, Getting Along With Others, and Maintaining Composure. Table 1 describes 
each of these skills and provides sample topics and lessons for each of the skills. Students from 
several schools within Region One in South Texas completed SEL lessons during Spring 2020 
and then completed a survey after each set of lessons. Lessons on Sustaining Effort were 
implemented first, in February and early March of 2020. The COVID-19 pandemic affected our 
ability to teach the remaining lessons on Getting Along With Others and Maintaining 
Composure. Nevertheless, smaller samples of students completed some programming on either 
Getting Along With Others or Maintaining Composure via virtual instruction. 


ACT Research & ACT’s Center for Equity in Learning | August 2021 5 


Table 1. Definition of Targeted Skills and Corresponding Lessons 


Skill Student behaviors that Examples of lesson DY -X-Yel df ol d(o) ame) m-1aM->.¢-111) ©) (mM (ct-s-Le) 8] 
demonstrate the skill Ke) e) Cex) 


Sustaining Diligence, effort, ° Persistence One activity stresses the importance of 
Effort organization, self-control, ° Goal setting goals to students and prompts them to set 
and compliance with the ° Planning their own academic and extracurricular 
rules. * Time management goals. 
Getting Collaboration, empathy, ° Perspective taking One activity illustrates the concept of 
Along With helpfulness, trust, and ° Expressing feelings individual differences, the dangers of labels, 
Others trustworthiness. * Communication and ways to appreciate diversity. 
* Dangers of labels 
Maintaining Stress management, ° Mindfulness One lesson introduces students to the 
Composure emotional regulation, a * Identifying emotions concept of mindfulness and gives 
positive response to ° Managing emotions instructions on how they can practice being 
setbacks, and poise. * Gratitude mindful. 


Participants and Method 


For the Sustaining Effort lessons, a total of 491 students from nine schools started the survey, 
and 440 (90%) of those who started the survey completed all closed-ended questions. For 
Getting Along With Others lessons, 239 students from five schools started the survey, and 211 
(88%) of those who started the survey completed all closed-ended questions. For Maintaining 
Composure lessons, 88 students from five schools started the survey, and 77 (88%) of those 
who started the survey completed all closed-ended questions. Students who partially completed 
the survey were included in the analyses. 


The survey for each of the target skills included questions asking for basic information including 
the student’s name and school. In addition, the surveys included the seven Likert-type questions 
shown in Figures 1-3. Students responded on a six-point scale (1 = disagree a lot and 6 = 
agree a lot). Next, each survey included a question asking, “Compared to what | knew about 
[each skill] previously, | feel like | now know...” Students answered this question on a five-point 
scale (1 = a lot less and 5 = a lot more). Further, each survey included an open-ended question 
asking students to indicate one thing they learned from the lessons. Finally, each survey 
included five example classification questions. For each question, students selected the 
example that best illustrated each of the skills out of a list of five examples. 


As mentioned previously, the COVID-19 pandemic disrupted the lessons. Lessons on 
Sustaining Effort were implemented first, and students received an average of 4.2 lessons and 
2.8 hours on content targeting this skill. Students completed a smaller amount of programming 
on either Getting Along With Others or Maintaining Composure. Specifically, students from 
schools that implemented lessons on Getting Along With Others completed an average of 3.6 
lessons and 2.2 hours, and those from schools that implemented lessons on Maintaining 
Composure completed an average of 2.8 lessons and 2.2 hours. 
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Level | Efficacy Evidence 


Figure 1 shows student reactions to lessons on each of the skills. Students generally indicated 
that they enjoyed the lessons. Across levels of agreement, the majority of students agreed that 
they enjoyed lessons on Sustaining Effort (84.5%), Getting Along With Others (91.9%), and 
Maintaining Composure (90.1%). Most students also felt that learning about each skill could 
help them succeed in school: Sustaining Effort, 88.1%; Getting Along With Others, 93.2%; and 
Maintaining Composure, 93.8%. They also thought working to develop each skill was important: 
Sustaining Effort, 90.6%; Getting Along With Others, 96.0%; and Maintaining Composure, 
91.4%. Further, results were consistent across the three skills, indicating that students reacted 
positively and saw value in lesson content on each skill. These results provide Level | efficacy 
data for SEL lessons on each of these skills. 


Level Il Efficacy Evidence 


Figure 2 shows items capturing gains in content knowledge resulting from these lessons. These 
questions are expected to tap student knowledge instead of behavior because they ask about 
student confidence performing a task versus asking about their actual experience with the task. 
Students who lack knowledge of a task will most likely provide less confident ratings. For 
instance, students who do not know the components of perseverance would be likely to report a 
lack of confidence in their ability to persevere on difficult tasks. Following the lessons on each 
skill, most students reported feeling they had acquired relevant content knowledge. Following 
lessons on Sustaining Effort, 85.5% of students agreed that they were confident they could 
persevere on difficult tasks, and 90.3% agreed that they could set goals, which are two 
important skills. Following lessons on Getting Along With Others, 94.5% of students agreed that 
they were confident they could work together with other students. Additionally, 95.5% of 
students agreed that they felt confident in their ability to demonstrate behaviors consistent with 
this skill. Following lessons on Maintaining Composure, 85.2% of students agreed that they 
were confident they could control their emotions, and 81.5% of students agreed that they were 
confident they could manage their stress levels, which are two important aspects of this skill. 


Compared to what students knew beforehand, most students also reported knowing more about 
each skill following the lessons. Students agreed they knew either a little or a lot more on 
Sustaining Effort (70.3%), Getting Along With Others (71.8%), and Maintaining Composure 
(70.9%). Consistent with their self-reported perceptions of learning, students identified correct 
examples of each of these skills at levels above that of chance. The percentage of students who 
correctly answered at least three of the five questions were as follows: Sustaining Effort 
(45.2%), Getting Along With Others (76.3%), and Maintaining Composure (59.7%). 


Students also responded to an open-ended question asking them one thing they learned from 
the lessons on each skill. Many students mentioned important themes from these lessons. For 
Sustaining Effort, student responses included persistence/resilience (n = 131), goal setting (n = 
72), and hard work (n = 34). For Getting Along With Others, student responses included the 
perceived value of teamwork such as getting things done better and faster (n = 71), working 
together (n = 32), and valuing different opinions (n = 20). For Maintaining Composure, student 
responses included emotional regulation (n = 27), stress management (n = 17), and self-control 
(n=9). 


Taken together, these results regarding students’ increased content knowledge provide Level II 
efficacy data for lessons targeting each of these skills. 
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Figure 1. Level | Efficacy Evidence for SEL Lessons on Each Skill 


Sustaining Effort Total 
Agreed 
Sustaining Effort. : Sth 
| think practicing Sustaining Effort é 
: : 3 88.1% 
will help me succeed in school. 
| think working to develop my 
Sustaining Effort skills is 90.6% 
important. 
0 20 40 60 80 100 
Getting Along With Others 
Soli anion ath ota A aaa «12 
Getting Along with Others. ps 
| think practicing Getting Along 
with Others will help me PSM 88.2% 
succeed in school. 
Getting Along with Others skills 
is important. 
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Maintaining Composure 
Maintaining Composure. 
| think practicing Maintaining 
in school. 
| think working to develop my 
Maintaining Composure sls is (Bi SE ox. 
important. 
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= Disagree a lot » Disagree = Disagree a little w Agree a little Agree a Agree a lot 


Note. For questions on Sustaining Effort, Ns ranged from 454 to 457. For questions on Getting 
Along With Others, Ns ranged from 220 to 223. For questions on Maintaining Composure, N = 
81. Values shown to the right of the graph denote % of respondents that reported some level of 


agreement with each question. 


Absence of a bar indicates 0 responses. 
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Figure 2. Level Il Efficacy Evidence for SEL Lessons on Each Skill 


Sustaining Effort 
persevere on difficult tasks. 
goals for myself. 
0 20 40 60 80 10 


Getting Along With Others 
| feel confident that | can work , 
together well with other students. 
| feel confident that | know what 
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Getting Along with Others. 
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Maintaining Composure 


| feel confident that! can control 
my emotions when | need to. 
| feel confident that | can manage 
my stress levels. 
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Note. For questions on Sustaining Effort, N = 454. For questions on Getting Along With Others, 
N = 220. For questions on Maintaining Composure, N = 81. Values shown to the right of the 
graph denote % of respondents that reported some level of agreement with each question. 


Absence of a bar indicates 0 responses. 
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Level Ill Efficacy Evidence 


Figure 3 shows items capturing student confidence in applying what they learned from these 
lessons both inside and outside school. For all skills, most students felt confident applying what 
they learned about each skill in school. Specifically, students agreed that in school they were 
applying knowledge from lessons on Sustaining Effort (78.8%), Getting Along With Others 
(92.4%), and Maintaining Composure (87.7%).Outside school, most students also agreed they 
were confident applying content on Sustaining Effort (78.3%), Getting Along With Others 
(89.7%), and Maintaining Composure (84.0%). These results provide initial Level III efficacy 
data for lessons on each of these skills. Given that these results are based on self-reported 
questions, an important next step for future research is using questions rated by others (e.g., 
teachers, parents) to further support this level of efficacy. 
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Figure 3. Level Ill Efficacy Evidence for SEL Lessons on Each Skill 


Sustaining Effort Total 


| am applying things | learned in 


projects | do in school. 
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Note. For questions on Sustaining Effort, N = 457. For questions on Getting Along With Others, 
N = 223. For questions on Maintaining Composure, N = 81. Values shown to the right of the 
graph denote % of respondents that reported some level of agreement with each question. 
Absence of a bar indicates 0 responses. 
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Conclusions 


Taken together, these findings provide efficacy evidence for SE skill-building lessons on 
Sustaining Effort, Getting Along With Others, and Maintaining Composure. In response to 
survey items measuring Kirkpatrick’s Levels | through III outcomes, students reacted positively 
to the lessons, demonstrated evidence of improved content knowledge on various measures, 
and reported confidence in applying content from the lessons both inside and outside school. 
Overall, these results provide some initial evidence consistent with Level III of Kirkpatrick’s 
model, but further evidence is needed to demonstrate changes in student behavior. Additional 
research is also needed to examine these lessons using a broader sample of historically 
underrepresented groups of students, using more rigorous research designs, and comparing the 
efficacy of SEL lessons delivered in-person versus via virtual instruction. Overall, these results 
provide support for the efficacy of the SE skill-building lessons for students from underserved 
communities, a group of students who often need additional supports because they face 
significant barriers to college entry and completion. Additionally, these results are promising 
given the challenges the COVID-19 pandemic caused during the administration of the lessons. 
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ACT’s Center for Equity in Learning focuses on closing gaps in equity, opportunity, and achievement 
for underserved populations and working learners. Through purposeful investments, employee 
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actionable research, initiatives, campaigns, and programs to further ACT’s mission of helping people 
achieve education and workplace success. 

For more information, visit equityinlearning.act.org 
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About ACT Research 


ACT Research leads the field with authority and high-quality scientific evidence in support of 
education and workforce practices, solutions, and services. Our mission-driven team comprises a 
variety of backgrounds and disciplines and offers a wide spectrum of knowledge and skills, enabling 
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Together, our research teams provide policymakers, educators, parents, and learners with research- 
based insights to inform their decision-making and deliver educators and workforce development 
professionals with tools and services needed for education and career navigation. 

For more information, visit www.act.org/research 
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NewSchools Venture Fund is a nonprofit venture philanthropy that invests in promising teams of 
educators and entrepreneurs with the vision and skills to reimagine learning. We help them 
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most ambitious dreams and plans. Through our investing, management assistance, network building 
and thought leadership, NewSchools helps to reimagine PreK-12 education. 

For more information, visit www.newschools.org 
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The Region One Education Service Center is part of a state-wide system of 20 regional education 
service centers created in 1965 by the 59th Texas Legislature to assist school districts across the 
state. Originally slated to work with school districts as a media center, the role of the education 
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implement initiatives assigned by the legislature or commissioner. Located in South Texas on the 
United States/Mexico border, Region One ESC serves 38 school districts and 10 charter school 
systems in the eight county areas of Brooks County, Cameron County, Hidalgo County, Jim Hogg 
County, Starr County, Webb County, Willacy County, and Zapata County. 

For more information, visit www.esc1.net/domain/3 


